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‘This post is one in a series highlighting MDRC’s methodological work. Contribu- 
tors discuss the refinement and practical use of research methods being employed 
across our organization. 


An earlier post in this series discussed considerations for reporting and inter- 
preting cross-site impact variation and for designing studies to investigate such 
cross-site variation. This post discusses how those ideas were applied to address 
two broad questions in the Mother and Infant Home Visiting Program Evalu- 
ation (MIHOPE): (1) whether variation in the way program services were im- 
plemented was related to variation in impacts across local programs, and (2) 
whether variation in the amount of services participants received was related to 
variation in impacts. Results from those analyses can be found in the MIHOPE 
15-month impact report." 


MIHOPE is the national evaluation of the Maternal, Infant, and Early Childhood 
Home Visiting (MIECHV) program. MIECHV was authorized by Congress in 
2010 and started a major expansion of evidence-based home visiting programs 
for families living in at-risk communities.” The study is being conducted for the 
U.S. Department of Health and Human Services by MDRC with several other 
research organizations. MIHOPE includes the four evidence-based models of 
home visiting that 10 or more states chose in their initial MIECHV plans in fis- 
cal year 2010-2011. To provide rigorous evidence on the MIECHV-funded pro- 
grams effects, the study randomly assigned about 4,200 families at 86 sites in 12 
states to either a program group who received MIECHV-funded home visiting 
or to a control group who received information on community services. 
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of a local program’s home visitors. To conduct this anal- 
ysis, the study used the method of Bloom, Hill, and Ric- 
cio (2003), conducting a multilevel regression where the 
explanatory variables included program and individual 
characteristics interacted with the treatment group indi- 
cator. The main concern in conducting the analysis was 
that local programs were not randomized to have different 
features, so results would not necessarily indicate causal 
relationships. For this reason, the results were interpreted 
as showing associations between features and impacts that 
might not be causal. 


Causal mediation analysis. In causal mediation analy- 
sis, the total effect of an intervention is divided into the 
direct effect of being assigned to receive the intervention 
and the indirect effect that operates through mediators: 
outcomes that might be affected by random assignment 
and in turn affect other outcomes — the receipt of pro- 
gram services, for example.? In essence, the indirect effect 
is estimated through a regression where the mediators are 
used as explanatory variables and the outcome is the in- 
dependent variable. The method relies on an assumption 
called “sequential ignorability.” Sequential ignorability 
has a few elements, but an essential one is that there are 
no unmeasured participant characteristics that confound 
the relationship between the mediator and the outcome 
once research group assignment (program or control) and 
participant characteristics are taken into account. In MI- 
HOPE, this assumption would be true only if the analysis 
included all participant characteristics and other charac- 
teristics that are correlated with both the mediator and the 
outcome. Even with the wealth of data collected for MI- 
HOPE, it was not possible to include all such characteris- 
tics. To reduce the consequences of violations of sequential 
ignorability, the main analysis in MIHOPE used a “control 
function” approach in which the residual from a regres- 
sion of the mediators on research group status was includ- 
ed as an explanatory variable in the regression in which 


= Instrumental variables. Instrumental variable analy- 


sis was developed for situations in which an explanatory 
variable and an outcome are correlated with a third, unob- 
served variable, confounding estimates of the relationship 
between the explanatory variable and the outcome. Instru- 
mental variable analysis relies on finding another variable 
— the instrument — that is correlated with the outcome 
and the explanatory variable but is not correlated with the 
third variable. In MIHOPE, instrumental variable analy- 
sis was used to examine the relationship between amounts 
of program services (such as the average number of home 
visits a program group member received) and impacts. 
Randomization provided the instrument, since whether 
someone was assigned to the program group or control 
group was associated with the home visiting services an 
individual received but not correlated with any potential 
confounding factors that existed before random assign- 
ment (for example, motivation to receive home visiting). 
In MIHOPE, this analysis boiled down to looking at the 
relationship by site between estimated impacts and the av- 
erage level of home visiting services received by program 
group families. For a multisite study with multiple medi- 
ators such as MIHOPE, Reardon and Raudenbush (2013) 
note that the method provides unbiased estimates only if 
cross-site variation in the impact on the mediators is not 
related to cross-site variation in the effect of the mediators 
on outcomes. This assumption would rule out (for exam- 
ple) having some sites with mothers who are both more 
motivated to receive home visiting and for whom home 
visiting is more or less effective. If their greater motivation 
is part of the reason their outcomes improve, results for 
those mothers could not be generalized to mothers with 
less motivation. Although this assumption is unlikely to 
be true for home visiting, the study team thought it was 
still informative to know whether local programs where 
families received more home visits had larger effects than 
those where families received fewer home visits.” 


the mediators were included as exploratory variables.* ‘The research team saw value in each of these approaches but 


also had concerns about the validity of each approach for the 


3 See, for example, Kosuke Imai, Luke Keele, Dustin Tingley, and Teppei Yamamoto, “Unpacking the Black Box of Causality: Learning About 
Causal Mechanisms from Experimental and Observational Studies” (American Political Science Review 105, 4: 765-789, 2011). 


4 This approach was originally suggested by Wooldridge (2015) and applied to causal mediation analysis by Courtemanche, Tchernis, and Ukert 
(2018). See Jeffrey Wooldridge, “Control Function Methods in Applied Econometrics” (Journal of Human Resources 50, 2: 420-445, 2015); 
Charles Courtemanche, Rusty Tchernis, and Benjamin Ukert, “The Effect of Smoking on Obesity: Evidence from a Randomized Trial” (Journal 
of Health Economics 57: 31-44, 2018). 


5 The assumption is unlikely to be true for home visiting because more motivated mothers are likely to have better outcomes even without home 
visiting, but are also likely to be more engaged in home visiting and receive more services. The relationship between the services received and 
outcomes for such mothers is because of the motivation, not the services received. 
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reasons outlined above. Meta-regression, instrumental vari- 
ables, and causal mediation analysis were chosen for several 
reasons. Each could simultaneously look at several continu- 
ous, explanatory factors (such as aspects of program imple- 
mentation for the meta-regression and types of home visit- 
ing services for instrumental variables and causal mediation 
analysis). The team also considered using two methods that 
rely on predicting who would be in a subgroup defined by a 
mediator, such as who would have received program services. 
The team decided not to use these methods in the published 
report because they were designed to look at a discrete num- 
ber of subgroups.° In addition, instrumental variable and 
causal mediation analysis provided alternative ways of ex- 
amining the relationship between program services and im- 
pacts, and the consistency of the results of the two analyses 
was one criterion used in deciding how much to emphasize 
specific findings. 


6 These methods are discussed in materials presented in September 2015 to the MIHOPE Advisory Committee. See “Mother and Infant 
Home Visiting Program Evaluation (MIHOPE): Investigating Variation in Program Impacts” (website: www.acf.hhs.gov/sites/default/files/ 
opre/5miechvpe_sac_sept_2015_mtg_impact_variation_memo_final_508.pdf, 2015). 
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